Systems and methods for use in detecting falls utilizing thermal sensing

ABSTRACT

Systems and methods designed to detect a human being falling as opposed to an inanimate object. Generally, the systems and methods will utilize a depth camera, which will often image in the NIR spectrum to detect a falling object. The portion detected as a falling object will often be detected as separating from a point cloud indicative of one object in contact with another. Should such a separation be detected, the systems and methods will utilize a thermal sensor, often a camera imaging in the LWIR spectrum, to determine if the falling portion has a heat signature indicative of a human being.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a Continuation of U.S. Utility patent applicationSer. No. 15/634,520, filed Jun. 27, 2017, and currently pending, andclaims benefit of U.S. Provisional Patent Application Ser. No.62/355,728 filed Jun. 28, 2016 and expired. The entire disclosure of allthe above documents is herein incorporated by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

This disclosure relates to systems for automatically assessing whetheran individual observed by the system has suffered a fall while they arewithin a particular environment. The disclosure particularly relates tousing thermal sensing in such a system to better segregate a fallingpatient from a falling non-patient object.

2. Description of the Related Art

Watching a toddler learn to walk, one may find it difficult to believethat falling can be one of the most dangerous things that can happen toa human being. While children are known to fall down on what seems to bea near constant basis and generally jump right back up, as one ages thepotential damage from a fall can go up dramatically.

When people discuss fall risk and the dangers of falls, they aregenerally talking about the risks for the elderly, which is a termcommonly used to refer to those over age 65. That population is oftenmuch more susceptible to body damage from a fall, more likely to beunable to get help for themselves after a fall, and is more likely tosuffer from falls as well. This population can be prone to increasedfalls from a myriad of problems such as worsening eyesight (e.g. due toissues such as presbyopia), decreases in muscle mass and reducedstrength, and from taking medications which may induce dizziness orvertigo. Further this population is often more susceptible to damagefrom falls due to weakening of bones and lack of musculature which meansthat an impact from a fall is more likely to do more serious damage.Finally, as many of them live alone or away from caregivers for extendedperiods of time, when they fall, they often cannot get to a phone orother external connection to get help.

Falls in the elderly can be a substantial problem. It has been estimatedthat falls are the leading cause of both fatal and nonfatal injuries inthe elderly and are one of the primary causes of broken bones(particularly hips) and head trauma. It has been estimated that 33% ofthe elderly will fall every year and that a third of those who fall willsuffer moderate to severe injuries (or even death) because of the fall.This means that those who house or serve the elderly on a regular basisneed to be constantly vigilant for the potential for falls and respondto them quickly so that any injuries incurred can be treated before theyare made worse by the passage of time.

Even outside of concerns about the elderly, falls can still present amajor concern. This is particularly true in medical and hospitalsettings. In these settings, even normally able-bodied people can besusceptible to a dramatically increased risk of falls and the elderly(who often require more medical attention) can be particularlysusceptible. Treatments and medications (most notably anesthetics andpain killers) used in medical settings can make patients dizzy,nauseous, or confused leading to them having a greatly heightened riskof falls. Further, injuries or symptoms which sent the person to thehospital in the first place (for example muscle weakness, damaged bones,or pain) can make a patient more susceptible to falls as well.

The susceptibility of the patient population to falls is also combinedwith institutional issues with hospitals and other medical facilitieswhich can increase fall risk and severity. Hospitals often have smoothsurfaced, and very hard, floors for easy cleaning and disinfection, butthis can also make them slippery and more likely to cause injury.Further, hospital equipment is often bulky, but needs to be placed inclose proximity to patient areas to make it accessible quickly which canreduce open areas and require more complicated navigation. Finally,since a hospital is generally a foreign environment to the patient, theyare also susceptible to simple lack of familiarity and can misestimatethe size and shape of steps or pathways resulting in a fall.

Falls for hospitalized patients are believed to present 30-40% of safetyincidents within any hospital and will generally occur at a rate of 4-14for every 1000 patient days at a hospital. For even a relatively smallfacility, this can lead to multiple fall incidents every month, and canmake them a near daily occurrence for a large institution. Whileinstitutions will typically utilize systems that allow them to try andreduce the number of falls that occur (for example, requiring patientsto be moved using wheelchairs), the fact that falls will occur to atleast some patients in a facility is unavoidable. By humans utilizingbipedal upright motion, some people will, in any given time window,suffer a fall.

Because of the fact that complete prevention of falls is essentially animpossible dream, there is a recognition that while a reduction in thenumber of falls is desirable, it is also important to make sure thatfalls are quickly responded to. Falling, particularly in aninstitutional setting, can often be an indicator of a secondary, andpotentially more serious, issue. While falls can be caused by simplemovement miscalculation (walking upright has actually been characterizedby some scientists as simply controlled falling), they can also becaused by loss of consciousness, dizziness, loss of motor control, orlack of strength which can be indicators of major medical conditionssuch as a stroke. Further, as a fall can result in further injury, it isdesirable to make sure that those injuries are quickly recognized andtreated. A patient who suffered a fall could, for example, tear stichesfrom prior surgery resulting in bleeding. This scenario is readilytreated if detected quickly, but it can be dangerous or deadly if not.In institutional settings where the population is often more susceptibleto falls and more likely to suffer injury from a fall, detecting that anindividual has suffered a fall so that aid can be provided to themquickly can be very important to mitigate the effects of the fall.

Because basic privacy concerns, and manpower issues, will generallyprevent institutional personnel from watching every patient all thetime, various automated systems have been proposed to try and bothassess fall risk and to detect falls. U.S. patent application Ser. No.13/871,816, the entire disclosure of which is herein incorporated byreference, provides for a system for fall detection and risk assessmentwhich externally analyzes gait parameters of a patient to evaluate boththeir likelihood of fall risk and to notify a caregiver if a fall isdetected.

The systems described in U.S. patent application Ser. No. 13/871,816utilize a depth camera or other device which can obtain depth image datato analyze an individual's gait. Image analysis, such as is described inthat application, effectively requires 3-Dimensional (“3D”) image data,which is why a depth camera is used. Image analysis can be very valuablein fall risk assessment as certain elements of gait, and changes ingait, can indicate increased likelihood of falling. Further, certainactions in a gait (such as the motion of stumbling) can be immediateindicators of a dramatically increased immediate fall risk or, uponanalysis of the 3D image data, that a fall has occurred or is occurring.Machines can generally automatically detect that such a fall hasoccurred based on the movement of the patient and immediately notifycaregivers to come to their aid.

Throughout this disclosure, it should be recognized that there aregenerally two different types of issues related to falls. A person'sfall risk is the likelihood that a person will fall at some time duringtheir stay in an institution. Generally, any person that can stand is ata non-zero fall risk as even completely able-bodied individuals can tripand fall unexpectedly. This application is not primarily concerned withdetermining fall risk and preventing falls. Instead, it is concernedwith detection that an individual has fallen so that aid can be providedto the fallen individual quickly.

To provide aid as quickly as possible after a fall and potentiallymitigate the effects from a fall, it is generally important that thecaregiver be notified that a patient has fallen very quickly (e.g. inreal-time or near real-time) after the person has fallen and the systemhas identified that a fall has occurred. Further, because of the natureof the notification, a caregiver will generally need to act quickly onthe notification, moving to the area where the patient is to assistthem. Because of this, it is extremely important that a system fordetecting falls not issue a large number of false positive detections.False positives can have the effect of “crying wolf” on the caregivers,and result in them not responding as quickly to an indication of apatient having fallen, resulting in a more negative outcome.

At the same time, a system for detecting falls is not particularlyvaluable if it generates false negatives. Where a patient has alreadyfallen, it is very important that the system detect this status quicklyand relay information that the fall has occurred to caregivers. If apatient falls and the fall is not detected close to the time the patientfalls, the patient may not be able to move in a fashion that the systemwould detect as a patient, and therefore the system may not detect thatthe patient needs care for a very long time which could result in a verydangerous situation. In effect, a patient that has already fallen isessentially a very low fall risk because they are generally not standingand, therefore, cannot fall. Thus, if the initial fall is not detectedas a fall, the system is unlikely to know to send caregivers at a latertime.

SUMMARY OF THE INVENTION

The following is a summary of the invention, which should provide to thereader a basic understanding of some aspects of the invention. Thissummary is not intended to identify critical elements of the inventionor in any way to delineate the scope of the invention. The sole purposeof this summary is to present in simplified text some aspects of theinvention as a prelude to the more detailed description presented below.

Because of these and other problems in the art, there is a need forimproved sensor systems for use in detecting falls that can eliminate orreduce either false negatives and/or false positives so as to make thesystem more accurate.

Described herein, among other things, is a method of determining if ahuman being has fallen and an associated system, the method comprising:providing a monitor including a depth camera and a thermal sensoroperatively coupled to a processor for interpreting the output of boththe depth camera and the thermal sensor; the depth camera imaging afirst point cloud; the depth camera imaging a separation of a portion ofthe first point cloud from the first point cloud; the processordetermining from the imaging of the separation that the portion'sseparation is indicative of the portion falling; the thermal sensorimaging heat emitted by the portion and the first point cloud; and theprocessor determining that a human being has fallen only if the portionincludes greater heat emitted than the first point cloud.

In an embodiment of the method and system, the depth camera is anear-infrared (NIR) camera.

In an embodiment of the method and system, the monitor also includes anNIR light source.

In an embodiment of the method and system, the thermnnal sensor is aninfrared camera.

In an embodiment of the method and system, the thermal sensor is along-wave infrared (LWIR) camera.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a general block diagram of an embodiment of a system fordetecting falls utilizing thermal sensing.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

Throughout this disclosure, certain terms will generally be consideredto have certain meaning. While not limiting the definition of theseterms as would be understood to one of ordinary skill, the following canassist in understanding the operation of the systems and methods.

The term “computer” as used herein describes hardware which generallyimplements functionality provided by digital computing technology,particularly computing functionality associated with microprocessors.The term “computer” is not intended to be limited to any specific typeof computing device, but it is intended to be inclusive of allcomputational devices including, but not limited to: processing devices,microprocessors, personal computers, desktop computers, laptopcomputers, workstations, terminals, servers, clients, portablecomputers, handheld computers, smart phones, tablet computers, mobiledevices, server farms, hardware appliances, minicomputers, mainframecomputers, video game consoles, handheld video game products, andwearable computing devices including but not limited to eyewear,wristwear, pendants, and clip-on devices.

As used herein, a “computer” is necessarily an abstraction of thefunctionality provided by a single computer device outfitted with thehardware and accessories typical of computers in a particular role. Byway of example and not limitation, the term “computer” in reference to alaptop computer would be understood by one of ordinary skill in the artto include the functionality provided by pointer-based input devices,such as a mouse or track pad, whereas the term “computer” used inreference to an enterprise-class server would be understood by one ofordinary skill in the art to include the functionality provided byredundant systems, such as RAID drives and dual power supplies.

It is also well known to those of ordinary skill in the art that thefunctionality of a single computer may be distributed across a number ofindividual machines. This distribution may be functional, as wherespecific machines perform specific tasks; or, balanced, as where eachmachine is capable of performing most or all functions of any othermachine and is assigned tasks based on its available resources at apoint in time. Thus, the term “computer” as used herein, can refer to asingle, standalone, self-contained device or to a plurality of machinesworking together or independently, including without limitation: anetwork server farm, “cloud” computing system, software-as-a-service, orother distributed or collaborative computer networks.

Those of ordinary skill in the art also appreciate that some deviceswhich are not conventionally thought of as “computers” neverthelessexhibit the characteristics of a “computer” in certain contexts. Wheresuch a device is performing the functions of a “computer” as describedherein, the term “computer” includes such devices to that extent.Devices of this type include but are not limited to: network hardware,print servers, file servers, NAS and SAN, load balancers, and any otherhardware capable of interacting with the systems and methods describedherein in the matter of a conventional “computer.”

For purposes of this disclosure, there will also be significantdiscussion of a special type of computer referred to as a “mobiledevice”. A mobile device may be, but is not limited to, a smart phone,tablet PC, e-reader, or any other type of mobile computer. Generallyspeaking, the mobile device is network-enabled and communicating with aserver system providing services over a telecommunication or otherinfrastructure network. A mobile device is essentially a mobilecomputer, but one which is commonly not associated with any particularlocation, is also commonly carried on a user's person, and usually is inconstant communication with a network.

Throughout this disclosure, the term “software” refers to code objects,program logic, command structures, data structures and definitions,source code, executable and/or binary files, machine code, object code,compiled libraries, implementations, algorithms, libraries, or anyinstruction or set of instructions capable of being executed by acomputer processor, or capable of being converted into a form capable ofbeing executed by a computer processor, including without limitationvirtual processors, or by the use of run-time environments, virtualmachines, and/or interpreters. Those of ordinary skill in the artrecognize that software can be wired or embedded into hardware,including without limitation onto a microchip, and still be considered“software” within the meaning of this disclosure. For purposes of thisdisclosure, software includes without limitation: instructions stored orstorable in RAM, ROM, flash memory BIOS, CMOS, mother and daughter boardcircuitry, hardware controllers, USB controllers or hosts, peripheraldevices and controllers, video cards, audio controllers, network cards,Bluetooth® and other wireless communication devices, virtual memory,storage devices and associated controllers, firmware, and devicedrivers. The systems and methods described here are contemplated to usecomputers and computer software typically stored in a computer- ormachine-readable storage medium or memory.

Throughout this disclosure, terms used herein to describe or referencemedia holding software, including without limitation terms such as“media,” “storage media,” and “memory,” may include or excludetransitory media such as signals and carrier waves.

Throughout this disclosure, the term “network” generally refers to avoice, data, or other telecommunications network over which computerscommunicate with each other. The term “server” generally refers to acomputer providing a service over a network, and a “client” generallyrefers to a computer accessing or using a service provided by a serverover a network. Those having ordinary skill in the art will appreciatethat the terms “server” and “client” may refer to hardware, software,and/or a combination of hardware and software, depending on context.Those having ordinary skill in the art will further appreciate that theterms “server” and “client” may refer to endpoints of a networkcommunication or network connection, including but not necessarilylimited to a network socket connection. Those having ordinary skill inthe art will further appreciate that a “server” may comprise a pluralityof software and/or hardware servers delivering a service or set ofservices. Those having ordinary skill in the art will further appreciatethat the term “host” may, in noun form, refer to an endpoint of anetwork communication or network (e.g. “a remote host”), or may, in verbform, refer to a server providing a service over a network (“hosts awebsite”), or an access point for a service over a network.

Throughout this disclosure, the term “real-time” generally refers tosoftware performance and/or response time within operational deadlinesthat are effectively generally cotemporaneous with a reference event inthe ordinary user perception of the passage of time for a particularoperational context. Those of ordinary skill in the art understand that“real-time” does not necessarily mean a system performs or respondsimmediately or instantaneously. For example, those having ordinary skillin the art understand that, where the operational context is a graphicaluser interface, “real-time” normally implies a response time of aboutone second of actual time for at least some manner of response from thesystem, with milliseconds or microseconds being preferable. However,those having ordinary skill in the art also understand that, under otheroperational contexts, a system operating in “real-time” may exhibitdelays longer than one second, such as where network operations areinvolved which may include multiple devices and/or additional processingon a particular device or between devices, or multiple point-to-pointround-trips for data exchange among devices. Those of ordinary skill inthe art will further understand the distinction between “real-time”performance by a computer system as compared to “real-time” performanceby a human or plurality of humans. Performance of certain methods orfunctions in real-time may be impossible for a human, but possible for acomputer. Even where a human or plurality of humans could eventuallyproduce the same or similar output as a computerized system, the amountof time required would render the output worthless or irrelevant becausethe time required is longer than how long a consumer of the output wouldwait for the output, or because the number and/or complexity of thecalculations, the commercial value of the output would be exceeded bythe cost of producing it.

As discussed herein, the system and methods generally are designed todetect falls from a patient in a hospital setting, a resident in asenior living community, or a person in a home setting. That is, theyoperate within a controlled environment and as such relate to detectinga fall while the patient is within that environment. While this is notrequired and any setting can utilize the systems and methods, thesesettings generally provide concerns for increased fall risk. Further,the systems and methods generally provide for the indication that a fallhas been detected to be relayed to a remote caregiver or monitor. Assuch, they are very useful for allowing a smaller number of personnel tomonitor a patient population generally considered to be at a heightenedrisk for falls and fall related injury.

The fall detection methods discussed herein are generally performed by acomputer system (10) such as that shown in the embodiment of FIG. 1. Thesystem (10) comprises a computer network which includes a central serversystem (301) serving information to a number of clients (401) which canbe accessed by users (501). The users are generally humans who arecapable of reacting to a fall as part of their job or task description.Thus, the users will commonly be medical personal, corporate officers,or risk management personnel associated with the enviromment beingmonitored, or even the patient themselves or family members orguardians.

In order to detect a fall, the server (301) will take in a variety ofinformation from a plurality of sensors (101) which will provide variousindications of the person's current actions. From those sensors' (101)output, a variety of characteristics considering if movement appears tocorrespond to a patient (201) falling can be determined. Thesecharacteristics may then be processed by the server (301) to produce adetermination if a fall has occurred. This determination is then passedon to the appropriate client(s) (401) and user(s) (501) for them toreact to.

The system (10) can also provide feedback to mobile devices (413), suchthe smartphone of a patient's doctor who may not be currently at acomputer. Similarly, information or requests for feedback may beprovided to a patient (201) directly. For example, if a patient (201) isdetected as having fallen, the system may activate a communicationsystem (415) in the patient's (201) room asking them to indicate if theyhave fallen. This can allow a patient (201) to rapidly cancel a falsealarm, or to confirm if they are in need of immediate aid.

An important aspect of the detection system (10) is that generally noneof the sensors (101) are tethered to the patient (201). That is that thepatient (201) does not need to wear any sensor or comply with anyprotocol for the fall to be detected. This allows for the system (10) tobe institutional and to monitor any, and generally all, patients (201)in the facility environment at all times. It also allows for the system(10) to not require the patient (201) to be setup on the system (10) inorder for it to begin monitoring. Instead, monitoring can begin of anyindividual as soon as they are present in the facility or in the sensingrange of the sensors (101). Still further, it eliminates concern of apatient (201) forgetting to take a monitor with them and rendering thesystem (10) impotent to detect them and even allows the system tomonitor those present if they are not patients (201), but are otherwisepresent in the facility.

The systems and methods discussed herein are designed to detect a humanbeing falling as opposed to an inanimate object. Generally, the systemsand methods will utilize a depth camera, which will often image in theNIR spectrum to detect a falling thing. The portion detected as afalling object will often be detected as separating from a point cloudindicative of one object in contact with another. Should such aseparation be detected, the systems and methods will utilize a thermalsensor, often a camera imaging in the LWIR spectrum, to determine if thefalling portion has a heat signature indicative of a human being. Thethermal sensor can also be used to determine if any object detected asfalling, even if not separating from another point cloud, by evaluatingthe heat signature to determine if it is indicative of a human.

FIG. 1 provides an overview of a system (10) and can be used toillustrate how the determination of a fall will generally work. Thefirst element is the sensor array (101) which is used to monitor thepatient (201). In order to untether the patient (201) from the system(10), these sensors (101) will generally be remote from the patient(201) (e.g. not located on their person or carried by them). Instead,they are generally located in areas where the patient (201) is likely tobe. The first, and generally primary, sensor is a depth camera (103) forcapturing depth image data.

In an embodiment, the depth camera (103) will comprise a camera whichtakes video or similar image-over-time data to capture depth image data.Specifically, this provides for 3D “point clouds” which arerepresentative of objects in the viewing range and angle of the camera(103). Operation of depth cameras (103) is generally well known to thoseof ordinary skill in the art and is also discussed in U.S. patentapplication Ser. No. 13/871,816, the entire disclosure of which isherein incorporated by reference, amongst other places. In order toprovide for increased privacy, the depth camera (103) may utilizesilhouette processing as discussed in U.S. patent application Ser. No.12/791,496, the entire disclosure of which is herein incorporated byreference. To deal with monitoring at night or under certain other lowconditions, the depth camera (103) may utilize recording optics forrecording the patient in an electromagnetic spectrum outside of humanvision. That is, the camera (103), in an embodiment, may record in theinfra-red or ultra-violet portions of the spectrum.

It is preferred that the camera (103) utilize an infrared (IR) sensitivecamera (and particularly a near-infrared (NIR) camera) utilizing activeimaging and an IR light source (113). This can allow for active imagingeven at night by providing an NIR light source (113) in the room andcollecting images in primarily the NIR band. As the NIR light (123) isnot detected by the human eye, the room is still dark to the patient(201) while the NIR camera (103) can still image clearly.

The camera (103) will generally utilize video or other multipleframe-over-time recording processes to search for patterns in motionthat can be indicative of a fall having previously occurred.Specifically, the camera (103) image will generally be processed toprovide for a variety of the elements and variables used in the falldetermination. Specifically, the camera (103) will generally beinterested in moving objects whose movement ceases or dramaticallychanges as these represent potential falls.

While the depth capturing camera (103) can operate in a variety of ways,in an embodiment the camera (103) will capture an image and theprocessor (311) will obtain the image, in real-time or near real-timefrom the camera (103) and begin to process the images. Initially,foreground objects, represented as a three dimensional (3D) point cloud(a set of points in three dimensional space), can be identified from thedepth image data using a dynamic background subtraction techniquefollowed by projection of the depth data to 3D. Generally, objects inthe foreground which are moving are considered to be of interest asthese can potentially represent a patient (201) in the room. In FIG. 1,the image includes two foreground objects: the patient (201) and a chair(203).

A tracking algorithm can then be used to track foreground objects overtime (generating a path history) and indicating that objects are moving.Walking sequences can then be identified from the path history of atracked object by identifying sections of the history during which theobject's movement met a set of criteria such as maintaining a minimumspeed for at least a minimum duration and covering at least a minimumdistance. Such walking sequences can then be processed to generatetemporal and spatial gait parameters, including walking speed, averagespeed, peak speed, stride time, stride length, and height, among others.As indicated above, U.S. patent application Ser. Nos. 12/791,496 and13/871,816 provide examples of how depth image information may beprocessed to provide gait and other stability information.Alternatively, objects (201) and (203) can simply be classified aseither moving or not moving across any particular number of framesrecorded by the camera (103).

One problem with detecting falls in hospital rooms, or in other roomswith beds, chairs (203), or similar pieces of furniture, is that thereare generally a number of objects on the furniture which can fall fromthe furniture and which may appear as a falling individual in collectedsensor data, including data from the camera (103), even if they are nota patient (201). This can include items such as pillows or blankets. Thefurniture itself may also move under certain circumstances. For example,chairs may roll on castors or curtains in the room may move in a breeze.Algorithms for detecting a patient (201) exiting a bed or chair (203)can incorrectly detect a fall if the patient (201) has been in the bedor chair (203) for a period of time so that in the depth camera (103)image they appear merged with surrounding objects and have now becomepart of the background information. Alternatively, the chair (203) andpatient (201) could have merged into a single foreground object due totheir proximity even if the patient has continued to be in motion. Thiscan make it difficult to detect which point cloud portion is the patient(201) versus the chair (203) or an object on the chair (such as ablanket) as the objects begin to separate when the user moves.

Objects merging into the background or each other in a depth camera(103) image is particularly problematic in fall detection because it isreasonably likely that an object, such as bedding, could fall from a bedwhile the patient (201) is still on the bed. As the camera (103)evaluates moving objects, an object which suddenly separates from thebed and falls to the floor can be detected as a falling patient (201).Similar problems can also occur where a privacy curtain is suddenlymoved by a breeze or where another object in the room suddenly hasmotion which then ceases. This latter case can occur, for example, if apatient left the viewing area but dropped a large package within theviewing area as they were doing so.

This type of false positive is particularly likely because thealgorithms for fall detection generally work backward from the finalfall position, to evaluate movement leading up to that position, todetennine if a fall has occurred. That is, the algorithms recognize thata moving object (point cloud) is no longer moving, and thus go backwardto evaluate the nature of the motion for a few seconds prior to themotion ceasing. If this movement is indicative of falling (e.g. it isdownward) this can trigger a fall determination. Thus, a falling pillowor blanket can greatly resemble a falling patient when one looks at thefinal position of the pillow (on the floor) and evaluates what it did inthe frames leading up to that point (fall from the bed to the floor).

Further, it can be difficult for the camera (103) to segregate a fallingpillow from a falling patient (201) because there is not necessarilyenough walking movement or other information prior to the fall for thecamera (103) to use walking behavior or other algorithms to evaluate thegeneral form and shape of the object to determine if it is a likelyhumanoid. The object became a foreground object of interest because itfell. Further, when methods such as those contemplated in U.S. patentapplication Ser. No. 12/791,496 for blurring of images and image edgesto maintain privacy are being used, depth images can have a hard timedetermining a rolled or balled human shape, versus it being the shape ofan inanimate object such as a pillow, or a combination of both.

The vast majority of these false positives in fall detection arebelieved to result from two specific facets of the fall detection. Thefirst is that the depth camera (103) image processing is generallylooking at objects which are moving, and which then stop moving (atleast in some dimensions or respects) in a position indicative of havingfallen to the floor to locate a fall. The system then goes backward fromthe fall position, to evaluate the movement to see if it is indicativeof a fall. Thus, an inanimate object in the frame which falls canprovide for a very likely false positive because this is precisely howthey move.

The second reason for false positives is that a patient (201) in theroom which is not moving (such as when they are asleep) generally needsto become part of the background. If the camera (103) and processor(311) evaluated every object in the frame which was once moving, itwould require a lot more computational power, but could also generate alarge number of false signals. An object, such as chair (103), movedinto the room should not be monitored simply because it once moved andthe computation will become bogged down in useless information. However,by allowing objects to become part of the background when they have notmoved in a certain amount of time, it now becomes possible for thepatient (201) to be lost in the image. This is most commonly because thepatient (201) is now in a position (such as in bed or sitting in a chair(203)) where they are not moving and their point cloud has merged withthe cloud of the object (203).

The merging of point clouds creates the problem with object merging inthe image. Effectively, the patient (201) and chair (203) are now asingle object to the depth camera (103) (or no object if they havebecome part of the background). When the patient (201) stands, thesingle object becomes two (or one with a part moving) and it isnecessary to quickly asses which is the patient (201) and which is ofchair (203). As many falls occur at the time that a stationary objectstarts moving (e.g. as the patient (201) stands from sitting) if themoving object (or object portion) is not quickly assessed as the patient(201), it is possible that the system (10) will not detect a patient(201) who stands and quickly falls. This creates a likely (and highlydangerous) false negative situation.

As indicated above, however, the problem is that a patient (201) thatstarts to stand from sitting and quickly falls, depending on the preciseangle of the chair (203), can look very similar to a blanket on thepatient's (201) legs slipping off and falling when a patient (201) turnsin their sleep. It is, thus, desirable to be able to determine where thepatient is within a merged point cloud where a portion of the cloudbeings moving.

In order to assist in segregating the patient image from surroundingobjects, the system (10) utilizes a long-wave infrared (LWIR) camera(109) in combination with depth camera (103). As LWIR cameras (109) cansee heat to a degree and can generally identify and separate a heatproducing object from one which does not produce heat, they can commonlyseparate out portions of the patient's (201) body as it will produceheat while surroundings will not. As opposed to NIR cameras, LWIRcameras (109) image the radiation of heat at essentially anytemperature. While heated objects can emit wavelengths in the NIR andvisible light spectrums, these heated objects often have to be heated togreater than 250° C. in order to be detected from emission in the NIR orvisible spectrums. For the purposes of detecting warm blooded animalsand many common everyday heat emitting objects, such as humans, thistemperature range cannot be used and it is necessary to detect heat morein the range of 0°-100° C.

LWIR cameras (109) are generally not capable of acting as a depth camera(103), however. Specifically, because LWIR cameras (109) visualize heatemission and reflectance of an object, the output of an LWIR camera(109) in determining the actual heat of the object generally requiressome knowledge of the material of which the object is made (since theobjects are not actually perfect black box emitters). Further, LWIRcameras (109) can have trouble generating depth information since theyare not utilizing reflected light, but generated heat.

The sitting scenario is particularly apt at showing the concern becausea patient (201) getting up or turning over may knock blankets or pillowsunto the floor. Thus, as the initial singular image cloud of the patient(201), chair (203), and blanket splits into multiple images, one or partof those images is seen as having fallen. The system (10) needs todetermine if the fallen object is the chair (203) tipping over, theblanket falling off the patient, or the patient (201). The NIR depthcamera (103) may be unable to determine which part of the point cloud(or which point cloud if they have actually separated) is the patientquickly and without more information as each point cloud could showaspects of both being and not being the patient (201).

To avoid these problems, the present system (10) includes use of anaccurate remote temperature (LWIR or thermal) sensor or camera (109).The thermal sensor (109) is tasked with quickly evaluating the pointcloud(s) to determine where the patient (201) is within the pointcloud(s) based on their heat emission. It is important that their heatemission is not so much their shape as is performed by the depth camera(103), but is simply an evaluation of their heat radiation pattern. Thethermal sensor (109) may be used in conjunction with a standard 3Dimaging camera in the visual spectrum (103), or with an NIR camera(103), or both. As opposed to an NIR camera (103), which in anembodiment of the present systems still utilizes a 3D depth image toevaluate the shape and movement of NIR emitting or reflecting objects,the temperature sensor (109) is tasked with simply evaluating thetemperature characteristics of the objects and will commonly be taskedwith evaluating relative temperature characteristics between objects.

Generally, the LWIR camera (109) will be co-located with, or very closeto the NIR depth camera (103). In this way, the LWIR image can besuperimposed onto each point cloud image generated by the NIR camera(103) providing parts of certain clouds with a heat signature. Becausehumans emit heat, bare skin will generally be visible to an LWIR camera(109). As most humans in their daily lives are not completely coveredwith clothing or other objects (which can block LWIR emissions if theyare loose fitting or newly put on), at least patches of LWIR emissionwill commonly be parts of the patient (201), and an entire patient maybe visible if they are wearing clothing that has been on sufficient timeto be warmed. These heated portions can be used to indicate that a cloudto which the heated element is at least partially superimposed is likelythat of a patient (201). In this way, it can be possible to treat animage which includes a human (even if it also includes another object)as including a human and thus being the location of patient (201).

This joint imaging can allow, for example, for the processor (311) todetermine that the point cloud of a chair (203) and patient (201)includes the patient (201) which is the element of interest. The factthat the cloud may also include the chair (203), is not important. Whatis important is that the cloud of interest includes the patient (201).By using thermal detection related to specific thermal characteristicsof a patient (201) versus other items in the room that do not emit LWIRenergy, the differences between a point cloud which includes the patient(201) versus one that does not can more rapidly be determined and thepatient (201) quickly detected from a merged cloud in the event of cloudseparation.

To illustrate the operation, let us assume that patient (201) has beensitting in chair (203) sufficiently long that the depth camera (103) hasa single point cloud of the two together at the instant something inthat point cloud starts moving. The movement causes the depth camera(103) to treat this cloud (or at least a portion of it) as an object ofinterest. The depth camera (103) also detects that the moving piece ofthe cloud has fallen to the floor. The question at this time for theprocessor (311) is if the moving piece of the point cloud (which may beits own point cloud) did or did not include the patient (201).Traditionally, this was all the information the processor (311) had towork with.

In the present system (10), if the falling portion of the point clouddoes not include a heat emitter, it is likely that the falling objectdetected is not patient (201). The evaluation is reinforced if a portionof the same cloud, or a nearby cloud, which did not move does include aheat emitter. This scenario implies that the patient (201) is still inthe chair (203) and the moving point cloud was an object previously inthe chair. Similarly, if the falling portion of the point cloud doesinclude a heat emitter, it is more likely that the patient (201) fellout of the chair (203). This is reinforced if there are no, or fewer,heat emitters in the remaining portion of the cloud than there wereprior to the motion being detected or if the heat emitters are generallynow closer to the floor than they were previously.

It should be recognized that in an embodiment, the task of the thermalsensor (109) can be performed by a camera (103) in addition togenerating depth image data by combining both sensors (101) in a commonhousing and using software to analyze the different incomingwavelengths. This allows for the hardware function of a separatetemperature sensor (109) to be implemented using appropriate controlsoftware operating on processor (311) and controlling the joint camera(103) and (109).

While the above is focused on the separation of a merged point cloudincluding the patient (201), it should be recognized that combining ofan image of the patient (201) with another object is also an element ofinterest. For example, draperies or a chair (203) can be objects that anindividual's 3D cloud image could merge and then separate from and thepatient's passage can cause these objects to move and that movement tothen cease. This could give a false indication of a fall as those movedobjects come to rest after the patient (201) passage. However, as theobjects will not emit heat due to the patient's passage, they willcommonly be rapidly ignored once the patient is sufficient distance fromthem to result in a separate point cloud. Similarly, should a patient(201) fall into another object such as chair (203), the resultant pointcloud may not look like the patient (201) has fallen if the whole cloudis analyzed for prior movement. However, as the portion of the cloudwhich includes the heat emitter can be separately considered, a fall canbe detected here as well.

While the invention has been disclosed in conjunction with a descriptionof certain embodiments, including those that are currently believed tobe the preferred embodiments, the detailed description is intended to beillustrative and should not be understood to limit the scope of thepresent disclosure. As would be understood by one of ordinary skill inthe art, embodiments other than those described in detail herein areencompassed by the present invention. Modifications and variations ofthe described embodiments may be made without departing from the spiritand scope of the invention.

It will further be understood that any of the ranges, values,properties, or characteristics given for any single component of thepresent disclosure can be used interchangeably with any ranges, values,properties, or characteristics given for any of the other components ofthe disclosure, where compatible, to form an embodiment having definedvalues for each of the components, as given herein throughout. Further,ranges provided for a genus or a category can also be applied to specieswithin the genus or members of the category unless otherwise noted.

1. A monitor for determining if a human being has fallen, the monitorcomprising: a means for imaging depth in a first point cloud and asecond point cloud comprising a separated portion of said first pointcloud; a means for interpreting output of both said depth camera andsaid thermal sensor and determining from said imaging of said secondpoint cloud that said second point cloud's motion is indicative of saidseparated portion falling; a means for imaging emitted heat from saidfirst point cloud and said second point cloud; a means for comparingemitted heat from said first point cloud to emitted heat from saidsecond point cloud; and a means for determining that a human being hasfallen only if said second point cloud includes greater heat emittedthan said first point cloud.
 2. The monitor of claim 1, wherein saidmeans for imaging depth comprises a depth camera.
 3. The monitor ofclaim 2, wherein said depth camera is a near-infrared (NIR) camera. 4.The monitor of claim 3, wherein said monitor also includes a means foremitting NIR light.
 5. The monitor of claim 4, wherein said means forimaging emitted heat comprises a thermal sensor.
 6. The monitor of claim5, wherein said thermal sensor is a long-wave infrared (LWIR) camera. 7.The monitor of claim 1, wherein said means for imaging heat is along-wave infrared (LWIR) camera.
 8. The monitor of claim 1, whereinsaid means for determining comprises a computer.
 9. A system fordetermining if a human being has fallen, said system comprising: asensor, said sensor not being in physical contact with said human being;a central server communicatively coupled to said sensor, said centralserver comprising a microprocessor and a non-transitorycomputer-readable medium having computer-executable program instructionsthereon which, when executed by said microprocessor, cause said centralserver to perform the steps of: receiving from said sensor data aboutsaid human being gathered by said sensor; extracting from said dataabout said human a plurality of data features; determining whether saidhuman has fallen, based at least in part on said plurality of datafeatures.
 10. The system of claim 9, wherein said sensor is one of aplurality of sensors.
 11. The system of claim 10, wherein saidcomputer-executable program instructions when executed by saidmicroprocessor, cause said central server to perform the steps of:receiving from said plurality of sensors, data about said human beinggathered by said plurality of sensors; extracting from said data aboutsaid human a plurality of data features; determining whether said humanhas fallen, based at least in part on said plurality of data featuresfrom all of said plurality of sensors.
 12. The system of claim 10,wherein said plurality of sensors comprises at least a depth camera witha thermal sensor.
 13. The system of claim 12, wherein said depth camerais a near-infrared (NIR) camera.
 14. The system of claim 13, whereinsaid depth camera includes an NIR light source.
 15. The system of claim12, wherein said thermal sensor is a long-wave infrared (LWIR) camera.